Critical Appraisal of Systematic Reviews With Costs and Cost-Effectiveness Outcomes: An ISPOR Good Practices Task Force Report

A systematic review (SR) can provide rigorous and complete evidence to support decision makers who consider both the effectiveness cost-effectiveness of health interventions. dramatic increase in published economic (HE) studies, more specifically cost has resulted consequent proliferation reviews with outcomes (SR-CCEO).1Luhnen M. Prediger B. Neugebauer E.A.M. Mathes T. Systematic evaluations: a structured analysis characteristics methods applied.Res Synth Methods. 2019; 10: 195-206Crossref PubMed Scopus (10) Google Scholar,2Mandrik O. Ekwunife O.I. Meheus F. et al.Systematic as "lens evidence": Determinants breast cancer screening.Cancer Med. 8: 7846-7858Crossref (5) Scholar First, such help indentify strenghts weaknesses HE modelling methodologies, data for inputs. Second, SR-CCEOs may be informative decisionmakers resource allocation decisions interventions, especially countries limited capacity technology assessment (HTA). For purpose this article, studies are defined analyzing costs healthcare including descriptions cost-of-illness (economic burden disease) studies. By cost-effectivenessstudies we mean full evaluations, cost-minimization, analysis, cost-utility cost-benefit cost-consequence analysis. Sometimes might based on an explicit comparison alternatives. However, it is challenging appropriately interpret owing their heterogeneity applied reporting, furthermore, variability clinical settings original they include. Methodologic guidance checklists that improve quality SRs or decrease risk bias interpretation synthesis3Shea B.J. Reeves B.C. Wells G. al.AMSTAR 2: critical appraisal tool include randomised non-randomised both.Bmj. 2017; 358: j4008Crossref (2521) Scholar, 4Oxman A.D. Guyatt G.H. Validation index articles.J Clin Epidemiol. 1991; 44: 1271-1278Abstract Full Text PDF (610) 5Whiting P. Savovic J. Higgins J.P. al.ROBIS: new assess was developed.J 2016; 69: 225-234Abstract (681) 6Moher D. Liberati A. Tetzlaff Altman D.G. Preferred reporting items meta-analyses: PRISMA statement.PLoS 2009; 6e1000097Crossref (39378) have applicability SR-CCEOs. There little specific methodologic SR-CCEOs.7Thielen F.W. Van Mastrigt Burgers L.T. al.How prepare evaluations practice guidelines: database selection search strategy development (part 2/3).Expert Rev Pharmacoecon Outcomes Res. 16: 705-721Crossref (53) 8van G.A. Hiligsmann Arts J.J. informing evidence-based decisions: five-step approach 1/3).Expert 689-704Crossref (88) 9Wijnen Redekop W.K. Majoie H. De Kinderen R. Evers S. How extraction, bias, transferability 3/3).Expert 723-732Crossref (85) 10Systematic reviewsCRD’s undertaking care. Center Reviews Dissemination York, 2009https://www.york.ac.uk/crd/SysRev/!SSL!/WebHelp/SysRev3.htmDate accessed: March 5, 2020Google 11Mathes Walgenbach Antoine S.L. Pieper Eikermann Methods review, comparison, synthesis method literature.Med Decis Making. 2014; 34: 826-840Crossref (15) Although Chapter 20 Cochrane Handbook Interventions Collaboration12Higgins J.P.T. Green Interventions. Version 6.0 [updated July 2019]. The Collaboration, 2019Crossref (24207) 3 articles related guidelines7Thielen guidance, recommendations do not focus evaluating conduct guidelines conducting identified multiple disagreements these recommendations, suggesting standardized needed.13Jacobsen E. Boyers Avenell Challenges Economic Evaluations: Review Recent Obesity Case Study.Pharmacoeconomics. 2020; 38: 259-267Crossref Making universal difficult because differ several important aspects, particular, regard inclusion criteria, types included (trial model-based, cost, cost-effectiveness), solely alongside outcomes. They also different objectives (eg, synthesize findings) identify gaps, used. Overall, SR-CCEO reliability usefulness will good objectives. Thus, ISPOR (The Professional Society Health Economics Research) established global, multistakeholder, multidisciplinary expert task force address need (Appendix 1 Supplemental Materials found at https://doi.org/10.1016/j.jval.2021.01.002). general provided, main goal regarding bias. This report, which includes Criteria Cost(-Effectiveness) (CiCERO) Checklist, assist researchers, producers technologies, users (decision makers/commissioners). categorized according 6 stages (Table 1).Table 1Overview major criteria outcomesNTitle stageTopics coveredStage 1Planning developmentClear objectivePredefined available protocolProtocol deviationsStage 2Search evidenceUpdate novel reviewComprehensive rapid reviewChoice database(s)No. databasesComprehensiveness reproducibilityUse supplementary materialsUse grey literatureStage 3Study eligibilityProcess study selectionEligibility usedStage 4Critical studiesTools appraise studiesProcess appraisalStage 5Data extraction synthesisProcess data-extractionAssessment heterogeneityMethods synthesisAssessment publication biasStage 6Presentation reportingReporting studiesReporting Open table tab Each SR should comprehensive predefined protocol. It preferred make protocol publicly prevent duplication ongoing reviews, reproducibility research, avoid selective reporting. achieved by registering either immediate delayed open access, (International prospective register PROSPERO, Centre Science, another independent online database), publishing it. Any deviations from final report publication. Independent availability, each clearly stated consistent its reported results conclusions, methods. routine develop eligibility around PICO (population, intervention, comparator, outcome) mnemonic reviews14Richardson W.S. Wilson M.C. Nishikawa Hayward R.S. well-built question: key decisions.ACP J Club. 1995; 123: A12-13Crossref evaluations.8van derivatives fully applicable appraising design models) illness) “comparator” “intervention” component absent. Depending SR-CCEO, focused on:•model-based studies: example, assessing models using life-time time horizon;•empirical treatment short horizon uses term empirical single study-based randomized nonrandomized trial-based observational [single arm, real-world data] used basis analyses, often called piggy-back Empirical contrasted modeling explicitly synthesizing various sources);•or both, broad perspectives horizons. Because inform makers, additional framing definitions essential: perspective. These elements define literature synthesis. cannot considered if through nontargeted, unsound, incomplete, nonreproducible search.15Levay Craven Searching. Practical Ideas Improving Results. Facet Publishing, London2019Google depends experience person group developed search.16Dickersin K. Scherer Lefebvre C. Identifying relevant reviews.BMJ. 1994; 309: 1286-1291Crossref (1414) Scholar,17Rethlefsen M.L. Farrell A.M. Osterhaus Trzasko L.C. Brigham T.J. Librarian co-authors correlated higher strategies internal medicine reviews.J 2015; 68: 617-626Abstract (158) Approaches improving involving information specialists library scientists peer-review electronic guidelines.17Rethlefsen Scholar,18McGowan Sampson Salzwedel D.M. Cogo Foerster V. PRESS peer strategies: 2015 guideline statement.J 75: 40-46Abstract (1280) If performed update existing reusing same appropriate. initial re-evaluated. answer amended research questions, reviewers ensure adaptations reflected strategy. Conducting time-consuming. takes average 17 months registered project start date.19Ganann Ciliska Thomas Expediting reviews: implications reviews.Implement Sci. 2010; 5: 56Crossref (416) Scholar,20Borah Brown A.W. Capers P.L. Kaiser K.A. Analysis workers needed medical interventions PROSPERO registry.BMJ Open. 7e012545Crossref (176) We expect similar timelines: adding words line result less hits, but complicated complementary needed. requires date within 12 date.12Higgins requirement appropriate summarizing Therefore, conducted shortest possible does compromise comprehensiveness updated before review’s narrowing SR-CCEO’s objective setting restrictions feasible defensible. believes duration crucial than other given Which sources justified primarily objectives, unlikely searching all literature.21Betran A.P. Say L. Gulmezoglu Allen Hampson Effectiveness databases identifying WHO maternal morbidity mortality.BMC Med Res Methodol. 2005; 6Crossref (54) viewpoints best search.7Thielen Scholar,22Wood Arber Glanville J.M. how extensive searches?.Int Technol Assess Care. 33: 25-31Crossref (9) Scholar,23Arber Isojarvi Baragula Edwards Shaw Wood evaluations?.Int 2018; 547-554Crossref (13) concluded Embase, HTA-journal database, MEDLINE/PubMed, enabled identification almost references SR-CCEO.23Arber To minimize missing recommend starting most commonly international (ie, umbrella review) showed resources (in order) were: MEDLINE, NHS EED (updated up 2015), checking reference lists, HTA databases22Wood (see Appendix B https://doi.org/10.1016/j.jval.2021.01.002 reflecting topics regional focus). Including likely comes records screening.24Hopewell McDonald Clarke Egger Grey meta-analyses trials care interventions.Cochrane Database Syst Rev. 2007; 18Mr000010Google Scholar,25Halladay C.W. Trikalinos T.A. Schmid I.T. C.H. Dahabreh I.J. Using beyond modest impact therapeutic interventions.J 1076-1084Abstract (93) least databases, chose to, well-justified confirmed evidence. enough reproducible, therefore, described detail. Existing filters studies.26Filters Identify EvaluationsISSG Search Filter Resource. InterTASC Information Specialists SubGroup.https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home?authuser=0Date November 2, In addition, filter (including Boolean operators), well considerations sensitivity specificity trade-offs useful.7Thielen Scholar,12Higgins authors whether applying (date publication, design, format, language, age subjects) limit literature. searched limits RCTs, misses possibly model-based research. Reviewers measuring cost/cost-effectiveness separate publications. outcome preferable results’ comprehensiveness. Even miss approximately 4% were missed searches.23Arber addition searches, “snowballing” techniques (searching bibliographies studies), personal knowledge citation tracking, contacting experts field.27Greenhalgh Peacock efficiency complex evidence: audit primary sources.BMJ. 331: 1064-1065Crossref (1017) means process supplemental searches28Canberra National Medical Research CouncilHow scientific series preparing guidelines.https://www.nhmrc.gov.au/about-us/publications/how-review-evidenceDate May 10, 1-step back tracking Searching dependent when conducted, potentially nonreproducible. refers unpublished been outside traditional commercial academic distribution channels. Examples government reports, policy statements, issues articles. particularly one way performed, justified. follow searches.29The Canadian Agency Drugs Technologies HealthGrey Matters: practical health-related literature.https://www.cadth.ca/resources/finding-evidence/grey-mattersDate August 28, reports B, sections 2 Furthermore, want explore platforms collect aggregate topics, Program Monitoring Emerging Diseases (ProMED) International Infectious (https://promedmail.org/about-promed/). As rule, abstracts conference proceedings search, even technically possible. Scientific could shown half ultimately fail publish after full,30Shemilt I. Khan N. Park Use compare reviews.Syst 140Crossref (64) Decision Making, Technology Assessment abstracts) indexed databases. Nevertheless, them solid argument inclusion, instance, further follow-up full-text Social networks (a social media website application sharing information) become unknown, seems obvious. apply derived without first screening titles, abstracts, promote transparency following guidelines, statement.6Moher “one-size-fits-all” approach, so evaluate contribute previously increases previous reviews’ steps applied. would only applied, due uncertainty steps. number tools SR-CCEO. AMSTAR-2 (A MeaSurement Tool Reviews) appraises selection,3Shea Robson al (2018) summarizes conclusions methods.31Robson R.C. Pham Hwee al.Few exist examining selecting abstracting data, review.J 106: 121-135Abstract (23) common recommendation excluding irrelevant study, perform step process, ideally independently, duplicate, conflicts resolved discussion third party while combination preferred. One limited, liberal reviewing titles reviewer then stage, there duplicate stringent application. mitigate any balance overinclusion (which costs) citations.30Shemilt Scholar,31Robson Another machine learning capabilities Abstrackr, DistillerSR, SWIFT-Active Screener, RobotAnalyst). manual selection. machine-learning time, currently uncertain. performance highly varied.32Gates Guitard Pillay al.AHRQ Effective Performance Usability Machine Learning Screening Reviews: Comparative Evaluation Three Tools. Healthcare Quality (US), Rockville (MD)2019Crossref 33Gartlehner Wagner Lux al.Assessing accuracy machine-assisted abstract DistillerAI: user study.Syst 277Crossref (11) 34Howard B.E. Phillips Tandon al.SWIFT-Active Screener: accelerated document active integrated recall estimation.Environment International. 138105623Crossref (26) nonvalidated artificial intelligence used, tested sample use reported. characterize greater relates relevance measures. restricting RCTs respect adverse event rates (underestimation), estimates effectiveness. SR-CCEOs, variety design. represents between validity broader generalizability (Box 1). perspective methodologies (how analyses) re